Journals
  Publication Years
  Keywords
Search within results Open Search
Please wait a minute...
For Selected: Toggle Thumbnails
MapReduce performance optimization based on anomaly detection model in heterogeneous cloud environment
HOU Jialin, WANG Jiajun, NIE Hongyu
Journal of Computer Applications    2015, 35 (9): 2476-2481.   DOI: 10.11772/j.issn.1001-9081.2015.09.2476
Abstract578)      PDF (788KB)(306)       Save
To effectively select the straggler machines, an anomaly detection model generally adopted in failure analysis was proposed. Firstly, an anomaly detection algorithm was employed to detect the slow nodes in the cluster. Secondly, task assignment algorithm and speculative execution algorithm were improved to stop assigning new tasks to slow nodes and these tasks were assigned to normal nodes with idle slots. In the improved speculative execution, it was for the first time that those tasks in slow nodes were transferred into the normal nodes in the same network segment, since data transferring can be physically accelerated in one network segment. The experimental results demonstrate that the straggler machines are quickly detected after running the anomaly detection algorithm. Compared with the algorithms in Hadoop-LATE, 17% of the processing time can be saved when the same amount of the tasks are processed, which concludes that the proposed algorithm is more suitable for improving the overall performance of the clusters.
Reference | Related Articles | Metrics